The Application of Information Integration Theory to Standard Setting: Setting Cut Scores Using Cognitive Theory

iv ACKNOWLEDGMENTS I would like to thank all the faculty members in the department for being patient with me and working hard to help me improve and mature. Specifically I would like to thank Craig Wells, who helped me with most of my projects and always gave encouraging words. Finally, I would like to thank the people at both HP and Excelsior College for their contributions to this work. Without their generosity, the topic of my dissertation would have been quite different. Information integration theory (IIT) is a cognitive psychology theory that is primarily concerned with understanding rater judgments and deriving quantitative values from rater expertise. Since standard setting is a process by which subject matter experts are asked to make expert judgment about test content, it is an ideal context for the application of information integration theory. Information integration theory (IIT) was proposed by Norman H. Anderson, a cognitive psychologist. It is a cognitive theory that is primarily concerned with how an individual integrates information from two or more stimuli to derive a quantitative value. The theory focuses on evaluating the unobservable psychological processes involved in making complex judgments. IIT is developed around four interlocking psychological The current study evaluates how IIT performs in an actual operational standard workshop across three different exams: HP storage solutions, Excelsior College nursing exam and the Trends for International Math and Science (TIMSS) exam. Each exam has cut scores set using both the modified Angoff method and the IIT method. Cut scores are evaluated based on Kane's (2001) framework for evaluating the validity of a cut score by evaluating procedural, internal and external sources of validity evidence. vi The procedural validity for both methods was relatively comparable. Both methods took approximately about the same amount of time to complete. Raters for both methods felt comfortable with the rating systems and expressed confidence in their ratings. Internal validity evidence was evaluated through the calculation of reliability coefficients. The inter-rater reliabilities for both methods were similar. However, the IIT method provided data to calculate intra-rater reliability as well. Finally, external validity evidence was collected on the TIMSS exam by comparing cut score classifications based on the Angoff and IIT methods to other performance criteria such as teacher expectations of the student. In each case, the IIT method was either equal or outperformed the Angoff method. Overall, the current study emphasizes the potential benefits IIT could …